This whole thing feels like more work than necessary. Let's just use Prometheus to record all this and then we can snapshot the DB when it's done. Maybe another option would work too, but let's not write our own custom metrics format and collection

Let's just use Prometheus to record all this and then we can snapshot the DB when it's done

Could you clarify a bit here?

AFAICT, we don't spin up a prometheus server to scrape/store our metrics. We just expose an endpoint for prometheus metrics to be scraped from. So there is no collection currently, except in-memory on the client.

These metrics are collected on the worker, and it isn't immediately obvious to me how we'd want to send worker metrics over to the client, to expose on its /metrics endpoint (if we want to do that at all), in tandem with running a prometheus server

So I opted to export this as a file

Maybe another option would work too, but let's not write our own custom metrics format

Yeah, fair. I can replace the current resource tracking with prometheus instrumentation.

What I mean is that we can just run a local Prom instance in-lieu of running inside actual infra. If/when we run inside full-on infra-provisioned stuff we can just use a real Prom service, but if we don't have one it's very easy to just spin one up locally and export the stuff from that.

We don't need to send metrics from the worker to the client. Both sides can just expose prom metrics for scraping independently and that's fine.